First commits to create cyhy-db #5

felddy · 2024-09-25T20:45:44Z

🗣 Description

This PR contains the initial functionality of the cyhy-db module, which provides a object model for accessing the various document collections in a Cyber Hygiene database.

This PR also includes an assortment of unit tests, though more testing will likely be added later.

💭 Motivation and context

We have two main motivations here:

To migrate from our legacy data model which was built on outdated Python 2 libraries to a more modern model that uses Python 3 (hello Beanie!)
To break the CyHy database model out into its own repository and module thus making it more testable and maintainable.

🧪 Testing

All unit tests currently pass in pytest. This code has also been partially tested by its use in the cyhy-kevsync module (initial PR coming soon).

✅ Pre-approval checklist

This PR has an informative and human-readable title.
Changes are limited to a single goal - eschew scope creep!
All future TODOs are captured in issues, which are referenced
in code comments.
All relevant type-of-change labels have been added.
I have read the CONTRIBUTING document.
These code changes follow cisagov code standards.
All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
Tests have been added and/or modified to cover the changes in this PR.
All new and existing tests pass.

✅ Pre-merge checklist

Finalize version.

✅ Post-merge checklist

Create a release.
Create (or move) v1 tag that points to new release.

Delete example package and files Update Python version to 3.12 in build workflows Update package name and description in setup.py Update dependencies in setup.py

Add comments to severity calculation.

This commit refactors the CVE model and test in the `cyhy_db` package. The changes include: - Replacing the `odmantic` library with `mongoengine` for the model definition. - Updating the field types and constraints in the CVE model. - Removing the `calculate_severity` method and integrating the severity calculation directly in the `save` method. - Updating the test cases to reflect the changes in the model.

…rsion

jsf9k

In addition to the quibbles below, when I run ./setup-env -p 3.13.0 -f I get an error:

src/cyhy_db/models/tally_doc.py:41: error: Incompatible types in assignment (expression has type "str", base class "Document" defined the type as "PydanticObjectId | None")  [assignment]

I did confirm that all tests pass by running pytest -vs --mongo-express.

.github/dependabot.yml

.github/workflows/build.yml

setup.py

src/cyhy_db/models/cve_doc.py

- Remove unnecessary pins. - Remove unused dependencies. - `coveralls` comment is no longer required. - `setuptools` pin and comment is no longer required. See: - cisagov/cyhy-config@8df2e69

Co-authored-by: David Redmin <[email protected]>

Add a mypy ignore directive to TallyDoc. Document all uses of this directive in a new issue.

felddy · 2024-10-15T17:45:47Z

In addition to the quibbles below, when I run ./setup-env -p 3.13.0 -f I get an error:
src/cyhy_db/models/tally_doc.py:41: error: Incompatible types in assignment (expression has type "str", base class "Document" defined the type as "PydanticObjectId | None")  [assignment]

I was able to replicate this. I've applied the same "fix" to TallyDoc that we used in the other model id fields. I've opened a new issue to track why this is the case, and added comments pointing to the issue.

See:

src/cyhy_db/models/place_doc.py

jsf9k · 2024-10-15T19:34:21Z

In addition to the quibbles below, when I run ./setup-env -p 3.13.0 -f I get an error:
src/cyhy_db/models/tally_doc.py:41: error: Incompatible types in assignment (expression has type "str", base class "Document" defined the type as "PydanticObjectId | None")  [assignment]
I was able to replicate this. I've applied the same "fix" to TallyDoc that we used in the other model id fields. I've opened a new issue to track why this is the case, and added comments pointing to the issue.

See:

5b2bb40

Revisit custom id field types and type: ignore[assignment] #7

Thank you! I was able to verify that I am now able to run ./setup-env -p 3.13.0 -f with no errors.

Add a mypy ignore directive to TallyDoc. Document all uses of this directive in a new issue.

dav3r

Strong work! 💪 💼

Please take a look at the couple of minor thangs I noticed.

src/cyhy_db/models/request_doc.py

src/cyhy_db/models/scan_doc.py

…-commits

The reserved word `from` was replaced with `from_` in `TicketDoc`. This follows that pattern. Co-authored-by: David Redmin <[email protected]> Co-authored-by: Jeremy Frasier <[email protected]>

This breaking change is documented in: - Data migration from `cyhy-core` database #8 Co-authored-by: David Redmin <[email protected]>

😜 Co-authored-by: dav3r <[email protected]>

….ini

I'm not sure if this is a work-around for a bug, or a needed option in Pydantic. See: - BeanieODM/beanie#369 - https://docs.pydantic.dev/latest/api/config/#pydantic.config.ConfigDict.populate_by_name

In an attempt to reach 100% coverage I broke out the various ways `reset_latestest_by_ip` could be called... to no avail. I believe this issue explains why the list comprehension shows a branch part as uncovered: - nedbat/coveragepy#1617

dav3r

👍 👍

felddy added 30 commits August 26, 2024 15:33

Add cyhy_db package and models

53be3c2

Delete example package and files Update Python version to 3.12 in build workflows Update package name and description in setup.py Update dependencies in setup.py

Apply changes made by linters

904692b

Add comments to severity calculation.

Add doc strings where needed

6593b87

Simplify field validation

312752b

Add test case for invalid CVSS score

250d6f6

Switch to MongoEngine

3f39aae

Add utcnow function to utils module

8999e26

Update MongoDB connection and add support for Mongo Express

3a291cd

Add IPAddressField and test cases for IP address validation and conve…

a568454

…rsion

Add ScanDoc and SnapshotDoc models

9c97a24

Sort imports

09fd02c

Implement CVE, ScanDoc with beanie

c29996e

Add cyhy_db package and initialize_db function

08920dd

Update conftest.py with Docker and mongo express fixtures

69117b4

Add test for database connection using motor and beanie libraries

4d620e6

Refactor imports and update test cases in test_cve.py

c7959a9

Remove test_ip_address.py

d551473

Add tests for ScanDoc model functionality

577ee7d

Add asyncio_mode to pytest.ini

f2e99a3

Switch to beanie ORM as a dependency

89e8303

Add enum classes for various types and statuses

2cf4587

Add RequestDoc model and test

43da94b

Refactor CVE model and add model validator for calculating severity

4162bdb

Rename ScanDoc test

c45dd48

Add RequestDoc model to ALL_MODELS list

a26b5c2

Add new dependencies to setup.py

11b5bb2

Add test case to reset latest flag by owner

1b1d518

Sort imports

2433f21

Refactor ScanDoc model and add new tests

7adf2d1

felddy requested a review from mcdonnnj as a code owner October 11, 2024 17:31

jsf9k requested changes Oct 11, 2024

View reviewed changes

.github/dependabot.yml Show resolved Hide resolved

.github/workflows/build.yml Outdated Show resolved Hide resolved

.github/workflows/build.yml Outdated Show resolved Hide resolved

setup.py Outdated Show resolved Hide resolved

Sort attributes in all document classes

1c53461

dav3r reviewed Oct 11, 2024

View reviewed changes

src/cyhy_db/models/cve_doc.py Outdated Show resolved Hide resolved

jsf9k mentioned this pull request Oct 11, 2024

First commits to create cyhy-kevsync cisagov/cyhy-kevsync#3

Merged

12 tasks

felddy and others added 4 commits October 11, 2024 17:09

Clean up dependencies

8fff2b4

- Remove unnecessary pins. - Remove unused dependencies. - `coveralls` comment is no longer required. - `setuptools` pin and comment is no longer required. See: - cisagov/cyhy-config@8df2e69

Replace magic string with proper enum

e5fdec2

Co-authored-by: David Redmin <[email protected]>

Revert hard coding of Python version

0f917ae

Work around type assignment error

5b2bb40

Add a mypy ignore directive to TallyDoc. Document all uses of this directive in a new issue.

dav3r reviewed Oct 15, 2024

View reviewed changes

src/cyhy_db/models/place_doc.py Outdated Show resolved Hide resolved

jsf9k approved these changes Oct 15, 2024

View reviewed changes

Work around type assignment error

fd82fe9

Add a mypy ignore directive to TallyDoc. Document all uses of this directive in a new issue.

felddy force-pushed the first-commits branch from 5b2bb40 to fd82fe9 Compare October 15, 2024 19:56

dav3r approved these changes Oct 15, 2024

View reviewed changes

src/cyhy_db/models/request_doc.py Outdated Show resolved Hide resolved

src/cyhy_db/models/scan_doc.py Outdated Show resolved Hide resolved

dav3r and others added 9 commits October 16, 2024 13:23

Merge branch 'first-commits' of github.com:cisagov/cyhy-db into first…

1d3cf32

…-commits

Rename reserved word deconfliction to match previously used pattern

07724b2

The reserved word `from` was replaced with `from_` in `TicketDoc`. This follows that pattern. Co-authored-by: David Redmin <[email protected]> Co-authored-by: Jeremy Frasier <[email protected]>

Remove field alias

438c199

This breaking change is documented in: - Data migration from `cyhy-core` database #8 Co-authored-by: David Redmin <[email protected]>

Remove inexplicable quoting

49af441

😜 Co-authored-by: dav3r <[email protected]>

Add asyncio_default_fixture_loop_scope configuration option to pytest…

a593d2d

….ini

Enable populate_by_name to handle aliases

86b493a

I'm not sure if this is a work-around for a bug, or a needed option in Pydantic. See: - BeanieODM/beanie#369 - https://docs.pydantic.dev/latest/api/config/#pydantic.config.ConfigDict.populate_by_name

Add ScanLimit save test

d338efe

Attempt to improve ScanDoc test coverage

84254b9

In an attempt to reach 100% coverage I broke out the various ways `reset_latestest_by_ip` could be called... to no avail. I believe this issue explains why the list comprehension shows a branch part as uncovered: - nedbat/coveragepy#1617

Sort pytest options

d0106dc

dav3r approved these changes Oct 21, 2024

View reviewed changes

felddy added 2 commits October 21, 2024 10:58

Update version file path in bump_version.sh

b1ffa0c

Bump version from 0.0.1 to 1.0.0

74a9505

felddy merged commit 3faa585 into develop Oct 21, 2024
23 checks passed

felddy deleted the first-commits branch October 21, 2024 15:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First commits to create cyhy-db #5

First commits to create cyhy-db #5

felddy commented Sep 25, 2024 •

edited

Loading

jsf9k left a comment •

edited

Loading

felddy commented Oct 15, 2024 •

edited

Loading

jsf9k commented Oct 15, 2024

dav3r left a comment

dav3r left a comment

First commits to create cyhy-db #5

First commits to create cyhy-db #5

Conversation

felddy commented Sep 25, 2024 • edited Loading

🗣 Description

💭 Motivation and context

🧪 Testing

✅ Pre-approval checklist

✅ Pre-merge checklist

✅ Post-merge checklist

jsf9k left a comment • edited Loading

Choose a reason for hiding this comment

felddy commented Oct 15, 2024 • edited Loading

jsf9k commented Oct 15, 2024

dav3r left a comment

Choose a reason for hiding this comment

dav3r left a comment

Choose a reason for hiding this comment

felddy commented Sep 25, 2024 •

edited

Loading

jsf9k left a comment •

edited

Loading

felddy commented Oct 15, 2024 •

edited

Loading